Surveys, Conjoint,
and Multilevel Models

IS 5150/6110

Surveys

An essential data collection method

Advantages

  • Easy to administer to a large number of respondents
  • Standardized questions provide a basis for comparison
  • Structured data simplifies analysis
  • Straightforward for respondents

Disadvantages

  • Getting the right sample can be difficult and expensive
  • Stuctured questions may unduly constrain respondents
  • Sensitive questions can be challenging to answer
  • Proper wording can be difficult

What makes a good survey?

  • Language and phrasing needs to fit the audience
  • Wording and structure should encourage completion
  • Make it easy to check any display or skip patterns
  • Language should be brief, clear, and conversational
  • Questions need to be answerable by the audience
  • Be careful about assumed knowledge and being ambiguous
  • Avoid double-barreled and leading/loaded questions

Survey tips

  • Use projection for potentially loaded questions
  • Make rating scales simple and anchored
  • Use display and skip patterns to funnel respondents to relevant questions
  • Include “Don’t know” or “None of the above” options
  • Include an “Other, please specify” open-ended option when you may not have exhaustive response categories
  • Consider randomizing question options to minimize order bias
  • Consider when to use forced choice

All measurements come with error

Ask easy questions first, difficult questions last

Conjoint

  • Products are defined by attributes, each with a number of levels
  • Respondents choose from among product alternatives
  • Respondent-level preferences are estimated for each attribute level
  • Preference estimates are used to make counterfactual predictions in a simulated market
  • Market simulators inform new product development, pricing, product line optimization, etc.

\[ U_{hj} = \beta_{h1}x_{j1} + \beta_{h2}x_{j2} + \cdots + \beta_{hk}x_{jk} + \epsilon_{hj} \]

\[ \color{grey}{U_{hj} = \color{black}{\beta_{h1}}x_{j1} + \color{black}{\beta_{h2}}x_{j2} + \cdots + \color{black}{\beta_{hk}}x_{jk} + \epsilon_{hj}} \]

\[ \color{grey}{U_{hj} = \beta_{h1}\color{black}{x_{j1}} + \beta_{h2}\color{black}{x_{j2}} + \cdots + \beta_{hk}\color{black}{x_{jk}} + \epsilon_{hj}} \]

Discrete Choice

For each respondent \(h\) and choice task \(t\) with \(j\) alternatives and \(k\) attribute levels:

\[ \begin{aligned} y_{ht} & = \text{argmax}(pr(y_{ht})), \ \text{s.t.} \ y_{ht} \in \left\{1, 2, \cdots, J \right\} \\[2mm] p(y_{htj}) & = {\exp\left(\beta_{h1}x_{tj1} + \beta_{h2}x_{tj2} + \cdots + \beta_{hk}x_{tjk} \right) \over \sum_{j=1}^J \exp\left(\beta_{h1}x_{tj1} + \beta_{h2}x_{tj2} + \cdots + \beta_{hk}x_{tjk} \right)} \\[2mm] \color{grey}{B} & \hspace{3mm} \color{grey}{\sim MVN\left(\gamma, \Sigma \right)} \\[2mm] \color{grey}{\gamma} & \hspace{3mm} \color{grey}{\sim \textit{Normal}(0, 1)} \\[2mm] \color{grey}{\Sigma} & \hspace{3mm} \color{grey}{= \text{diag}(\tau) \ \Omega \ \text{diag}(\tau)} \\[2mm] \color{grey}{\Omega} & \hspace{3mm} \color{grey}{\sim LKJ(1)} \\[2mm] \color{grey}{\tau} & \hspace{3mm} \color{grey}{\sim \textit{Half-Normal}(1, 2)} \end{aligned} \]

Discrete Choice

For each respondent \(h\) and choice task \(t\) with \(j\) alternatives and \(k\) attribute levels:

\[ \begin{aligned} \color{grey}{y_{ht}} & \hspace{3mm} \color{grey}{= \text{argmax}(pr(y_{ht})), \ \text{s.t.} \ y_{ht} \in \left\{1, 2, \cdots, J \right\}} \\[2mm] \color{grey}{p(y_{htj})} & \hspace{3mm} \color{grey}{= {\exp\left(\beta_{h1}x_{tj1} + \beta_{h2}x_{tj2} + \cdots + \beta_{hk}x_{tjk} \right) \over \sum_{j=1}^J \exp\left(\beta_{h1}x_{tj1} + \beta_{h2}x_{tj2} + \cdots + \beta_{hk}x_{tjk} \right)}} \\[2mm] B & \sim MVN\left(\gamma, \Sigma \right) \\[2mm] \color{grey}{\gamma} & \hspace{3mm} \color{grey}{\sim \textit{Normal}(0, 1)} \\[2mm] \color{grey}{\Sigma} & \hspace{3mm} \color{grey}{= \text{diag}(\tau) \ \Omega \ \text{diag}(\tau)} \\[2mm] \color{grey}{\Omega} & \hspace{3mm} \color{grey}{\sim LKJ(1)} \\[2mm] \color{grey}{\tau} & \hspace{3mm} \color{grey}{\sim \textit{Half-Normal}(1, 2)} \end{aligned} \]

\[ \LARGE{p(\theta | X) \propto p(X | \theta) \ p(\theta)} \]

\[ \LARGE{p(\theta, \alpha | X) \propto p(X | \theta) \ \color{red}{p(\theta | \alpha)} \ p(\alpha)} \]

Discrete Choice

For each respondent \(h\) and choice task \(t\) with \(j\) alternatives and \(k\) attribute levels:

\[ \begin{aligned} \color{grey}{y_{ht}} & \hspace{3mm} \color{grey}{= \text{argmax}(pr(y_{ht})), \ \text{s.t.} \ y_{ht} \in \left\{1, 2, \cdots, J \right\}} \\[2mm] \color{grey}{p(y_{htj})} & \hspace{3mm} \color{grey}{= {\exp\left(\beta_{h1}x_{tj1} + \beta_{h2}x_{tj2} + \cdots + \beta_{hk}x_{tjk} \right) \over \sum_{j=1}^J \exp\left(\beta_{h1}x_{tj1} + \beta_{h2}x_{tj2} + \cdots + \beta_{hk}x_{tjk} \right)}} \\[2mm] \color{grey}{B} & \hspace{3mm} \color{grey}{\sim MVN\left(\gamma, \Sigma \right)} \\[2mm] \gamma & \sim \textit{Normal}(0, 1) \\[2mm] \Sigma & = \text{diag}(\tau) \ \Omega \ \text{diag}(\tau) \\[2mm] \Omega & \sim LKJ(1) \\[2mm] \tau & \sim \textit{Half-Normal}(1, 2) \end{aligned} \]

discover.sawtoothsoftware.com

Multilevel Models

\[ \LARGE{p(\theta, \alpha | X) \propto p(X | \theta) \ p(\theta | \alpha) \ p(\alpha)} \]

Adaptive Shrinkage

import numpy as np
import polars as pl
import pymc as pm
import arviz as az

# Import (standardized) fox data.
foxes = pl.read_csv('../data/foxes.csv')

# Separate predictors and the outcome.
X = foxes.select(pl.col(['avgfood', 'groupsize'])).to_numpy()
y = foxes.select(pl.col('weight')).to_numpy().flatten()

# Estimate the direct causal effect of avgfood on weight.
with pm.Model() as foxes_model:
  # Data.
  X_data = pm.Data('X_data', X)
  y_data = pm.Data('y_data', y)

  # Priors.
  alpha = pm.Normal('alpha', mu = 0, sigma = 0.2)
  beta = pm.Normal('beta', mu = 0, sigma = 0.5, shape = 2)
  sigma = pm.Exponential('sigma', lam = 1)

  # Likelihood.
  mu = alpha + X_data @ beta
  y_obs = pm.Normal('y_obs', mu = mu, sigma = sigma, observed = y_data)

# Sample.
with foxes_model:
  draws = pm.sample()

# Visualize marginal posteriors.
az.plot_forest(draws, var_names=['beta'], combined = True, hdi_prob=0.95)

import numpy as np
import polars as pl
import bambi as bmb
import arviz as az

# Import foxes data.
foxes = pl.read_csv('../data/foxes.csv')

# Use Bambi to estimate the direct causal effect of avgfood on weight.
bambi_model_01 = bmb.Model('weight ~ avgfood + groupsize', foxes.to_pandas())
bambi_model_01
       Formula: weight ~ avgfood + groupsize
        Family: gaussian
          Link: mu = identity
  Observations: 116
        Priors: 
    target = mu
        Common-level effects
            Intercept ~ Normal(mu: -0.0, sigma: 2.4892)
            avgfood ~ Normal(mu: 0.0, sigma: 2.5)
            groupsize ~ Normal(mu: 0.0, sigma: 2.5)
        
        Auxiliary parameters
            sigma ~ HalfStudentT(nu: 4.0, sigma: 0.9957)

# Calls pm.sample().
bambi_fit_01 = bambi_model_01.fit()
az.plot_trace(bambi_fit_01, compact = False)

# Visualize marginal posteriors.
az.plot_forest(bambi_fit_01, var_names = ['avgfood', 'groupsize'], combined = True, hdi_prob = 0.95)

# Use Bambi to estimate the direct causal effect of avgfood on weight by group.
bambi_model_02 = bmb.Model('weight ~ (avgfood|group) + (groupsize|group)', foxes.to_pandas(), noncentered = True)
bambi_model_02
       Formula: weight ~ (avgfood|group) + (groupsize|group)
        Family: gaussian
          Link: mu = identity
  Observations: 116
        Priors: 
    target = mu
        Common-level effects
            Intercept ~ Normal(mu: -0.0, sigma: 2.4892)
        
        Group-level effects
            1|group ~ Normal(mu: 0.0, sigma: HalfNormal(sigma: 2.4892))
            avgfood|group ~ Normal(mu: 0.0, sigma: HalfNormal(sigma: 2.5))
            groupsize|group ~ Normal(mu: 0.0, sigma: HalfNormal(sigma: 2.5))
        
        Auxiliary parameters
            sigma ~ HalfStudentT(nu: 4.0, sigma: 0.9957)

# Calls pm.sample().
bambi_fit_02 = bambi_model_02.fit()
az.plot_trace(bambi_fit_02, compact = False)

# Visualize marginal posteriors.
az.plot_forest(bambi_fit_02, var_names = ['avgfood|group', 'groupsize|group'], combined = True, hdi_prob = 0.95)